Optical image stabilization movement to create a super-resolution image of a scene

ABSTRACT

The present disclosure describes systems and techniques directed to optical image stabilization movement to create a super-resolution image of a scene. The systems and techniques include a user device (102) introducing (502), through an optical image stabilization system (114), movement to one or more components of a camera system (112) of the user device (102). The user device (102) then captures (504) respective and multiple frames (306) of an image of a scene, where the respective and multiple frames (306) of the image of the scene have respective, sub-pixel offsets of the image of the scene across the multiple frames (306) as a result of the introduced movement to the one or more components of the camera system (112). The user device (102) performs (506), based on the respective, sub-pixel offsets of the image of the scene across the respective, multiple frames (306), super-resolution computations and creates (508) the super-resolution image of the scene based on the super-resolution computations.

BACKGROUND

Camera technology associated with a user device today, such as a digitallens single-reflex (DLSR) camera or a smartphone having an image sensor,has advanced on several fronts. As one example, the user device mayinclude an Optical Image Stabilization (OIS) camera module that combinesa camera system having the image sensor with an OIS system to compensatefor motion of the user device while the user device is capturing images.As another example, the user device may include super-resolutioncapabilities that create a super-resolution image from multiple,low-resolution images of a scene, wherein the multiple, low-resolutionimages of the scene are multiple images captured using native resolutioncapabilities of the camera system and the super-resolution image is of aresolution that is higher than the native resolution capabilities of thecamera system.

In some instances, use of the OIS system is incompatible with, or inconflict with, the super-resolution capabilities of the user device. Forexample, if the multiple, low-resolution images of a scene need toreflect position offsets (or differences in pixel positions) of theimages of the scene for super-resolution algorithms to function,stability introduced by the OIS system during the capture of themultiple, low-resolution images of the scene may prevent the neededposition offsets. As another example, if the user device is in a stablestate with no movement (and the OIS system is not functioning orcompensating for motion during the capture of the multiple-lowresolution images), the multiple low-resolution images may be identicaland not possess the needed position offsets.

Furthermore, the user device may capture the multiple images of thescene in a variety of modes, including a burst sequence mode that occursbefore and after the press of a shutter button, as well as during a zeroshutter-lag mode that occurs nearly simultaneously with the press of theshutter button. The aforementioned incompatibilities between the OISsystem and the super-resolution capabilities of the user device compoundthemselves further in such instances.

SUMMARY

The present disclosure describes systems and techniques directed tooptical image stabilization movement to create a super-resolution imageof a scene. The systems and techniques include a user device using anoptical image stabilization system to introduce movement to one or morecomponents of a camera system of the user device. The user device thencaptures respective and multiple frames of an image of a scene, wherethe respective and multiple frames of the image of the scene haverespective, sub-pixel offsets of the image of the scene across themultiple frames as a result of the introduced movement to the one ormore components of the camera system. The user device performs, based onthe respective, sub-pixel offsets of the image of the scene across therespective, multiple frames, super-resolution computations and createsthe super-resolution image of the scene based on the super-resolutioncomputations.

The system and techniques can therefore create super-resolution imageswithout affecting Optical Image Stabilization (OIS) system performance.Furthermore, the systems and techniques can stabilize the one or morecomponents of the camera system without distracting the user, work with“zero shutter-lag” modes, and don't require waiting for a user to pressthe shutter movement, minimizing input lag and latencies that may bedetrimental to obtaining high-quality images and to the user'sexperience.

In some aspects, a method used to create a super-resolution image of ascene is described. The method includes a user device introducingmovement to one or more components of a camera system of the userdevice. As part of the method, the user device captures respective andmultiple frames of an image of a scene, where the respective andmultiple frames of the image of the scene have respective, sub-pixeloffsets of the image of the scene across the multiple frames as a resultof the introduced movement to the one or more components of the camerasystem. The method continues, where, based on the respective, sub-pixeloffsets of the image of the scene across the respective, multipleframes, the user device performs super-resolution computations andcreates the super-resolution image of the scene based on thesuper-resolution computations.

In yet other aspects, a user device is described. The user deviceincludes a camera system that has image sensors and an OIS system,processors, and a display. The user device also includes acomputer-readable storage media storing instructions of an opticalimage-stabilization system drive manager application and asuper-resolution manager application that, when executed by theprocessors, perform complementary functions that direct the user deviceto perform a series of operations.

The series of operations includes receiving, by the one or moreprocessors, a command that directs the user device to capture an imageof a scene. The series of operations also includes introducing movementto one or more components of the camera system during the capture of theimage of the scene, wherein the introduced movement results in thecapture of respective and multiple frames of the image of the scene thathave respective, sub-pixel offsets of the image across the multipleframes and performing, by the one or more processors and based on therespective, sub-pixel offsets of the image of the scene across themultiple frames, super-resolution computations. The series of operationsfurther includes creating, by the one or more processors based on thesuper-resolution computations, a super-resolution image of the scene andrendering, by the display, the super-resolution image of the scene.

The details of one or more implementations are set forth in theaccompanying drawings and the following description. Other features andadvantages will be apparent from the description and drawings, and fromthe claims. This summary is provided to introduce subject matter that isfurther described in the Detailed Description and Drawings. Accordingly,a reader should not consider the summary to describe essential featuresnor limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

This present disclosure describes details of one or more aspectsassociated with optical image stabilization movement to create asuper-resolution image of a scene.

FIG. 1 illustrates an example operating environment in which variousaspects of optical image stabilization movement to create asuper-resolution image scene are performed.

FIG. 2 illustrates example aspects of movement introduced by an OISsystem.

FIG. 3 illustrates example aspects associated with detecting a motioncondition and introducing movement to an OIS system.

FIG. 4 illustrates example aspects of using multiple frames havingsub-pixel offsets to perform super-resolution computations and create asuper-resolution image of a scene.

FIG. 5 illustrates an example method used for optical imagestabilization movement to create a super-resolution image of a scene.

DETAILED DESCRIPTION

The present disclosure describes systems and techniques directed tooptical image stabilization movement to create a super-resolution imageof a scene.

While features and concepts of the described systems and techniques foroptical image stabilization movement to create a super-resolution imageof a scene can be implemented in any number of different environments,systems, devices, and/or various configurations, aspects are describedin the context of the following example devices, systems, andconfigurations.

Example Operating Environment

FIG. 1 illustrates an example operating environment 100 in which variousaspects of optical image stabilization movement to create asuper-resolution image of a scene are performed. As illustrated, a userdevice 102 is fixed to a tripod 104 and rendering a super-resolutionimage 106 of a scene that it is capturing. Although illustrated as asmartphone, the user device 102 may be another type of device that hasimage-capture capabilities, such as a DLSR camera or a tablet. Thefixing of the user device 102 to the tripod 104 constrains the userdevice 102 such that the user device 102 is stationary with no motion.

The user device 102 includes a combination of one or more motionsensor(s) 108 (e.g., a gyroscope, an accelerometer) that detect a motioncondition of the user device 102. In some instances, such as when theuser device 102 is fixed to the tripod 104, the detected motioncondition may be a static motion condition (i.e., the user device 102not moving). In other instances, such as when the user device 102 isremoved from the tripod 104, the detected motion condition may be adynamic motion condition (i.e., any motion condition that is not astatic motion condition).

The user device also includes an Optical Image Stabilization (OIS)camera module 110, which includes a camera system 112 and an OIS system114. The camera system 112 may include multiple components such as alens, an aperture, and one or more image sensors (e.g., 40 megapixel(MP), 32 MP, 16 MP, 8 MP). The image sensor(s) may include acomplementary metal-oxide semiconductor (CMOS) image sensor or acharge-coupled device (CCD) image sensor. In some instances, the imagesensor(s) may include a color filter array (CFA) that overlays pixels ofthe image sensor and limits intensities, as associated with colorwavelengths, of light recorded through the pixels. An example of such aCFA is a Bayer CFA, which filters light according to a red wavelength, ablue wavelength, and a green wavelength.

The OIS system 114, in general, provides mechanics to modify physicalpositions or orientations of one or more components of the camera system112. The OIS system 114 may include micro-scale motors andmagnetic-induction positioning mechanisms that can change an in-planeposition, an out-of-plane position, a pitch, a yaw, or a tilt of one ormore components of the camera system 112. The in-plane position may be aposition that lies within a plane defined by two axes, while theout-of-plane position may be another position that lies outside theplane defined by the two axes.

The user device 102 also includes a combination of one or moreprocessor(s) 116. The processor 116 may be a single core processor or amultiple core processor composed of a variety of materials, such assilicon, polysilicon, high-K dielectric, copper, and so on. In aninstance of multiples of the processor 116 (e.g., a combination of morethan one processor), the multiples of processor 116 may include acentral processing unit (CPU), a graphics processing unit (GPU), adigital signal processor (DSP), or an image processing unit (IPU).Furthermore, and in such an instance, the multiples of the processor 116may perform two or more computing operations using pipeline-processing.

The user device 102 also includes computer-readable storage media (CRM)118 that includes executable instructions in the form of an OIS drivemanager 120 and a super-resolution manager 122. The CRM 118 describedherein excludes propagating signals. The CRM 118 may include anysuitable memory or storage device such as random-access memory (RAM),static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM (NVRAM),read-only memory (ROM), or Flash memory useable to store the OIS drivemanager 120 and the super-resolution manager 122.

In some instances, the OIS drive manager 120 and the super-resolutionmanager 122 may be executed by the processor 116 to cause the userdevice 102 to perform complementary functions that are effective toproduce OIS movement (e.g., a physical movement or change to positionsof one or more elements of the camera system 112), capture multipleframes of an image of a scene with sub-pixel offsets, and compute thesuper-resolution image (106) of a scene. In some instances, the OISmovement may correspond to changes in position to the camera system 112that would manifest through a natural, handheld motion of a useroperating the user device (e.g., a hand tremor or other naturalbiomechanics of the user).

The code or instructions of the OIS drive manager 120 may be executed,using the processor 116, to direct the OIS system 114 to perform the OISmovement. In some instances, the OIS movement may be an unsynchronizedmovement (e.g., movement within a predetermined range of positionaldisplacements and having no temporal synchronization). Such anunsynchronized movement can be applicable to a static motion condition.

In other instances, the OIS movement may be a synchronized movement(e.g., movement corresponding to a range of positional displacementsthat are dependent upon, and temporally synchronized with, detectedmotions of the user device 102). Such a synchronized movement can beapplicable to a dynamic motion condition and superimposed onto anotherOIS movement introduced to the OIS system 114.

The code or instructions of the super-resolution manager 122, whenexecuted by the processor 116, may direct the user device 102 to performmultiple operations that are complementary to the operations performedunder the direction of the OIS drive manager 120. Such operations mayinclude directing the camera system 112 to capture, in a burst sequence,multiple frames of an image of a scene, perform super-resolutioncomputations, and render, using a display 124 of the user device, thesuper-resolution image 106 of the scene.

Due to the OIS movement (e.g., the movement introduced to the camerasystem 112 by the OIS system 114 under the direction of the OIS drivemanager 120), the camera system 112 may capture multiple variations 126,128, and 130 of the image of the scene, where the multiple variations126-130 correspond to multiple frames of the image of the scene thathave respective, sub-pixel offsets of the image across the multipleframes.

During the OIS movement, the OIS drive manager 120 may compute coveragescores, either analytically or numerically through uniform sampling ofthe multiple frames of the image of the scene. The computed coveragescores may, in some instances, cause additional or supplemental OISmovements to define image content of the captured, multiple variations126-130. This movement may ensure full coverage of the image of thescene and produce a higher quality image.

The super-resolution computations may include, for example, Gaussianradial basis function (RBF) computations combined with robustness modelcomputations to create the super-resolution image 106 of the scene. Thesuper-resolution computations use the variations 126-130 of the image ofthe scene (e.g., the multiple frames of the image of the scene that haverespective, sub-pixel offsets of the image across the multiple frames)to create the super-resolution image 106 of the scene.

FIG. 2 illustrates example aspects 200 of movement introduced by an OISsystem. Aspects 200 include the camera system 112 and the OIS system 114of the user device 102 of FIG. 1 .

As illustrated, the OIS system 114 may introduce movement to the camerasystem 112. An “in-plane” movement may be a movement contained within aplane defined by the x-axis 202 and the y-axis 204. The plane defined bythe x-axis 202 and the y-axis 204, corresponds to (e.g., is parallel to)another plane having the super-resolution image 106 of the scene. An“out-of-plane” movement may be a movement that is outside the planedefined by the x-axis 202 and the y-axis 204 and may be a movement alongthe z-axis 206 or another movement that includes a pitch, yaw, or roll.

The movement introduced by the OIS system 114 may be triggered by one ormore factors. For example, the movement introduced to the OIS system 114may in response to the user device 102 entering a viewfinder mode, inresponse to the user device 102 receiving a capture command, or inresponse to the user device 102 detecting a motion condition through themotion sensors 108. The movement introduced by the OIS system 114 mayalso be triggered through a combination of one or more of such factors.

FIG. 3 illustrates example aspects 300 associated with detecting amotion condition and introducing movement to an OIS system. The exampleaspects 300 may be performed by the user device 102 of FIG. 1 and theOIS camera module 110 of FIG. 2 . The OIS drive manager 120 (beingexecuted by the processor 116) may introduce movement to camera system112 through the OIS system 114. In some instances, the movementintroduced to the camera system 112 through the OIS system 114 maysimply stabilize the camera system 112 during the capture of an image ofa scene, while in other instances the movement introduced to the camerasystem 112 through the OIS system 114 may introduce sub-pixel offsets tomultiple images of a scene during the capture of the multiple images ofthe scene. In yet other instances, the movement introduced to the camerasystem 112 through the OIS system 114 may be a combination of movementsthat stabilize capture of some frames of the image the scene (e.g., nosub-pixel offsets for clarity purposes) and also introduce offsetsduring capturing of other frames of the image of the scene (e.g.,preserve sub-pixel offsets for super-resolution purposes).

The example aspects 300 include detection of a motion condition 302, anintroduction of movement 304, and a capturing of multiple frames 306having sub-pixel offsets (frames 308-312 correspond to variations126-128 of FIG. 1 ) The multiple frames 306 serve as a basis forcomputing and forming the super-resolution image 106. The user device102 may capture the multiple frames 306, using a resolution that islower than another resolution of the super-resolution image 106 of thescene, during a burst sequence.

Detection of the motion condition 302 may include detecting a staticmotion condition 314 or a dynamic motion condition 318. Introduction ofthe movement 304 to the camera system 112 may include the OIS drivemanager 120 introducing, during the burst sequence, an unsynchronizedOIS movement 316 or a synchronized OIS movement 320.

In the instance of the synchronized OIS movement 320, the OIS system 114may introduce a first movement that is intended to stabilize the camerasystem 112 from the detected dynamic motion condition 318 (e.g.,stabilize the camera system 112 to capture frames of an image of a scenewith clarity and without sub-pixel offsets). The synchronized movement320 may be a second movement that is synchronized with, and superimposedonto, the first movement (e.g., a movement that is opposite movementassociated with the detected dynamic motion condition 318) with theintent of generating or preserving sub-pixel offsets across the framesof the image of the scene. Depending on the magnitude of the firstmovement, the superimposing of the second movement may be constructiveor destructive in nature (e.g., “add to” or “subtract from” the firstmovement). Furthermore, some portions of the second movement may begreater than a pixel while other portions of the second movement may beless than a pixel.

The burst sequence may include capturing the multiple frames 306 at aset time interval that may range, for example, from one millisecond tothree milliseconds, one millisecond to five milliseconds, or one-halfmillisecond to ten milliseconds. Furthermore, and in some instances, thetime interval of the burst sequence may be variable based on a motion ofthe user device (e.g., a time interval may be “shorter” during ahigh-velocity motion of the user device 102 than another time intervalduring a low-velocity motion of the user device 102 to keep offsets atless than one pixel).

The introduction of the movement 304 during the burst sequenceeffectuates the user device 102 capturing the multiple frames 306 suchthat the multiple frames 306 have respective, relative sub-pixeloffsets. As illustrated the image of frame 310 is respectively offset,relative to the image of frame 308, one half-pixel horizontally and onehalf-pixel vertically. Furthermore, and as illustrated, the image offrame 312 is respectively offset, relative to the image of frame 308,one-quarter pixel horizontally. Respective, relative sub-pixel offsetscan include different magnitudes and combinations of sub-pixel offsets(e.g., one sub-pixel offset associated with one frame might beone-quarter pixel horizontally and three-quarters of a pixel vertically,while another sub-pixel offset that is associated with another framemight be zero pixels horizontally and one-half of a pixel vertically).In general, the techniques and systems described by this presentdisclosure can accommodate sub-pixel offsets that are more random thanthe illustrations and descriptions of frames 308-312, includingsub-pixel offsets that are non-uniform.

FIG. 4 illustrates example aspects 400 of using multiple frames 306having sub-pixel offsets, as introduced by OIS movement, to performsuper-resolution computations and create a super-resolution image of ascene. The example aspects 400 may use elements of FIGS. 1-3 , whereinperforming the super-resolution computations are performed by the userdevice 102 of FIG. 1 , the sub-pixel offsets are a result of the OISmovement of FIG. 2 , and the multiple frames having the sub-pixeloffsets are the multiple frames 306 of FIG. 3 .

As illustrated in FIG. 4 , the multiple frames 306 are input tosuper-resolution computations 402 performed by the user device 102.Algorithms that support the super-resolution computations 402 may residein the super-resolution manager 122 of the user device 102. In someinstances, the user device 102 (e.g., multiples of the processor 116)may perform portions of the super-resolution computations 402 usingpipeline-processing. In general, the super-resolution computations 402may use a variety of algorithms and techniques.

A first example of the super-resolution computations 402 includesGaussian radial basis function (RBF) kernel computations. To perform theGaussian RBF kernel computations, the user device 102 filters pixelsignals from each frame of the multiple frames 306 to generaterespective color-specific image planes corresponding to color channels.The user device 102 then aligns the respective color-specific imageplanes to a selected reference frame.

Continuing with the first example of the super-resolution computations402, the Gaussian RBF kernel computations can include the user device102 analyzing local gradient structure tensors to compute a covariancematrix. Computing the covariance matrix may rely on the followingmathematical relationship:

$\begin{matrix}{\Omega = {\left\lbrack {e_{1}\mspace{14mu} e_{2}} \right\rbrack\begin{bmatrix}k_{1} & 0 \\0 & k_{2}\end{bmatrix}}} & (1)\end{matrix}$In mathematical relationship (1), Ω represents a kernel covariancematrix, e₁ and e₂ represent orthogonal direction vectors and twoassociated eigenvalues λ₁ and λ₂, and k₁ and k₂ control a desired kernelvariance.

Computing the local gradient structure tensors may rely on the followingmathematical relationship:

$\begin{matrix}{\hat{\Omega} = \begin{bmatrix}I_{x}^{2} & {I_{x}I_{y}} \\{I_{x}I_{y}} & I_{y}^{2}\end{bmatrix}} & (2)\end{matrix}$In mathematical relationship (2), I_(x) and I_(y) represent local imagegradient in horizontal and vertical directions, respectively.

Also, as part of the first example of the super-resolution computations402, the user device 102 may compute a robustness model using astatistical neighborhood model that includes color mean and spatialstandard deviation computations. The robustness model computations, insome instances, may include denoising computations to compensate forcolor differences.

The super-resolution image computations 402 are effective to estimate,for each of the multiple frames 306 (e.g., for frame 308, 310, and 312),the contribution of pixels to color channels associated with respectivecolor planes

Continuing with the first example of the super-resolution computations402, color planes may be accumulated using the following mathematicalrelationship for normalization computations:

$\begin{matrix}{{C\left( {x,y} \right)} = \frac{\sum\limits_{n}\;{\sum\limits_{i}\;{c_{n,i} \cdot w_{n,i} \cdot {\hat{R}}_{n}}}}{\sum\limits_{n}\;{\sum\limits_{i}{w_{n,i} \cdot {\hat{R}}_{n}}}}} & (3)\end{matrix}$In mathematical relationship (3), x and y represent pixel coordinates,the sum Σ_(n) operates over (or is a sum of) contributing frames, thesum Σ_(i) is a sum of samples within a local neighborhood, c_(n,i)represents a value of a Bayer pixel at a given frame n and samplew_(n,i) represents a local sample weight, and {circumflex over (R)}_(n)represents a local robustness. The accumulated color planes may becombined to create the super-resolution image 106 of the scene.

A second example of the super-resolution computations 402 may includeanalyzing the influence of motion blur. Such super-resolutioncomputations may analyze the influence of motion blur of multiple images(e.g., the multiple frames 306) using a “volume of solutions”computational approach that addresses uncertainties in pixelmeasurements due to quantization errors.

A third example of the super-resolution computations 402 may includecomputations that use a frame-recurrent approach. Such an approach mayuse, iteratively, a previous low-resolution frame, a previously computedhigh-resolution image, and a current low-resolution frame to create acurrent super-resolution image (e.g., the previous low-resolution frameand the current low-resolution frame may be frame 308 and frame 310,respectively). The recurrent approach may include flow estimations toestimate a normalized, low-resolution flow map, upscaling thelow-resolution flow map with a scaling factor to produce ahigh-resolution flow map, using the high-resolution flow map to warp aprevious high-resolution image, mapping the warped, previoushigh-resolution image to a low-resolution space, and concatenatingmapping of the low-resolution space to the current super-resolutionimage.

The super-resolution computations 402 of FIG. 4 may also use algorithmsand techniques that are other or additional to those described in theprevious examples. Such algorithms and techniques may includemachine-learned algorithms and techniques. Regardless, and in accordancewith the present disclosure, introducing movement to the one or morecomponents of the camera system 112 are applicable to the otheralgorithms and techniques.

Example Method

Example method 500 is described with reference to FIG. 5 in accordancewith one or more aspects associated with optical image stabilizationmovement to create a super-resolution image of a scene. Generally, anyof the components, modules, methods, and operations described herein canbe implemented using software, firmware, hardware (e.g., fixed logiccircuitry), manual processing, or any combination thereof. Someoperations of the example methods may be described in the generalcontext of executable instructions stored on computer-readable storagememory that is local and/or remote to a computer processing system, andimplementations can include software applications, programs, functions,and the like. Alternatively or in addition, any of the functionalitydescribed herein can be performed, at least in part, by one or morehardware logic components, such as, and without limitation,Field-programmable Gate Arrays (FPGAs), Application-Specific IntegratedCircuits (ASICs), Application-Specific Standard Products (ASSPs),System-on-a-Chip systems (SoCs), or Complex Programmable Logic Devices(CPLDs).

FIG. 5 illustrates example aspects of a method 500 used as part ofcreating a super-resolution image of a scene. The method 500 isdescribed in the form of a set of blocks 502-508 that specify operationsthat can be performed. However, operations are not necessarily limitedto the order shown in FIG. 5 or described herein, for the operations maybe implemented in alternative orders, in fully or partially overlappingmanners, or in iterative fashions. Furthermore, although the operationsrepresented by the method 500 will be described in the context of beingperformed by the user device 102 of FIG. 1 using elements of FIGS. 2-4 ,the operations (or portions of the operations) may be performed by oneor more other devices having computational capabilities, such as aserver or a cloud-computing device including instructions (or portionsof instructions) of the super-resolution manager 122.

At block 502, the user device 102 (e.g., processor 116 executing thecode of the OIS drive manager 120) introduces movement to one or morecomponents of a camera system 112 (e.g., the OIS system 114 introducesthe movement to the camera system 112). In some instances, the userdevice 102 may introduce the movement in response to entering aviewfinder mode, while in other instances the user device 102 mayintroduce the movement in response to receiving a capture commandIntroducing the movement may include introducing an in-plane movement orintroducing an out-of-plane movement to the one or more components ofthe camera system 112.

At block 504, the user device 102 captures respective and multipleframes 306 of an image of a scene that, as a result of the introducedmovement to the one or more components of the camera system 112, haverespective, sub-pixel offsets of the image of the scene 306 across themultiple frames.

At block 506, the user device 102 (e.g., the processor 116 executing theinstructions of the super-resolution manager 122) performssuper-resolution computations 402 based on the respective, sub-pixeloffsets of the image of the scene across the multiple frames. Examplesof super-resolution computations 402 include (i) computing Gaussianradial basis function kernels and computing a robustness model, (ii)analyzing the influence of motion blur across the multiple frames, and(iii) using a frame-recurrent approach that uses, from the multipleframes, a previous low-resolution frame and a current low-resolutionframe to create a current super-resolution image.

At block 508, and based on the super-resolution computations 402, theuser device 102 (e.g., the processor 116 executing the instructions ofthe super-resolution manager 122) creates the super-resolution image 106of the scene.

The example method 500 may also include altering the movement based on acoverage score that is computed through a sampling of the multipleframes of the image of the scene.

Although the present disclosure describes systems and techniquesdirected to optical image stabilization movement to create asuper-resolution image of a scene, it is to be understood that thesubject of the appended claims is not necessarily limited to thespecific features or methods described. Rather, the specific featuresand methods are disclosed as example ways in which use of optical imagestabilization movement to create a super-resolution image of a scene canbe implemented.

Further to the descriptions above, a user may be provided with controlsallowing the user to make an election as to both if and when systems,programs or features described herein may enable collection of userinformation (e.g., images captured by a user, super-resolution imagescomputed by a system, information about a user's social network, socialactions or activities, profession, a user's preferences, or a user'scurrent location), and if the user is sent content or communicationsfrom a server. In addition, certain data may be treated in one or moreways before it is stored or used, so that personally identifiableinformation is removed. For example, a user's identity may be treated sothat no personally identifiable information can be determined for theuser, or a user's geographic location may be generalized where locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular location of a user cannot be determined. Thus, theuser may have control over what information is collected about the user,how that information is used, and what information is provided to theuser.

In the following, several examples are described:

Example 1: A method to create a super-resolution image of a scene, themethod performed by a user device and comprising: introducing, throughan optical image stabilization system, movement to one or morecomponents of a camera system of the user device; capturing respectiveand multiple frames of the image of the scene, the respective andmultiple frames of the image of the scene having respective, sub-pixeloffsets of the image of the scene across the multiple frames as a resultof the introduced movement to the one or more components of the camerasystem; performing, based on the respective, sub-pixel offsets of theimage of the scene across the multiple frames, super-resolutioncomputations; and creating, based on the super-resolution computations,the super-resolution image of the scene.

Example 2. The method as recited by example 1, wherein introducing themovement to the one or more components of the camera system is inresponse to entering a viewfinder mode.

Example 3: The method as recited by example 1, wherein introducing themovement to the one or more components of the camera system is inresponse to receiving a capture command.

Example 4: The method as recited by any of examples 1-3, whereinintroducing the movement to the one or more components of the camerasystem of the user device includes introducing an in-plane movement tothe one or more components of the camera system, wherein the in-planemovement is a movement contained in a plane defined by two axes.

Example 5: The method as recited by any of examples 1-4, whereinintroducing the movement to the one or more components of the camerasystem of the user device includes introducing an out-of-plane movementto the one or more components of the camera system, wherein theout-of-plane movement is a movement outside a plane defined by two axes.

Example 6: The method as recited by any of examples 1-5, whereinperforming the super-resolution computations includes: computingGaussian radial basis function kernels; and computing a robustnessmodel.

Example 7: The method as recited by any of examples 1-6, whereinperforming the super-resolution computations includes analyzing theinfluence of motion blur across the multiple frames.

Example 8: The method as recited by any of examples 1-7, whereinperforming the super-resolution uses a frame-recurrent approach thatuses, from the multiple frames, a previous low-resolution frame and acurrent low-resolution frame to create a current super-resolution image.

Example 9: The method as recited by any of examples 1-8, whereinadditional or supplemental movement is introduced to the one or morecomponents of the camera system based on a coverage score that iscomputed through a sampling of the multiple frames of the image of thescene.

Example 10: A user device comprising: a camera system; an optical imagestabilization system; one or more processors; a display; and acomputer-readable storage media storing instructions of an optical imagestabilization system drive manager application and a super-resolutionmanager application that, when executed by the one or more processors,perform complementary functions that direct the user device to: receive,by the one or more processors, a command that directs the user deviceto: capture an image of a scene; introduce, based on the receivedcommand and through the optical image stabilization system, movement toone or more components of the camera system during the capture of theimage of the scene, wherein the introduced movement results in thecapture of respective and multiple frames of the image of the scene thathave respective, sub-pixel offsets of the image across the multipleframes; perform, by the one or more processors and based on therespective, sub-pixel offsets of the image of the scene across themultiple frames, super-resolution computations; create, by the one ormore processors based on the super-resolution computations, asuper-resolution image of the scene; and render, by the display, thesuper-resolution image of the scene.

Example 11: The user device as recited by example 10, wherein the userdevice further comprises one or more motion sensors that detect a staticmotion condition.

Example 12. The user device as recited by example 11, wherein thedetection of the static motion condition causes the user device tointroduce, through the optical image stabilization system and to one ormore components of the camera system, an unsynchronized movement.

Example 13: The user device as recited by example 10, wherein the userdevice further comprises one or more motion sensors that detect adynamic motion condition.

Example 14. The user device as recited by example 13, wherein thedetection of the dynamic motion condition causes the user device tointroduce, through the optical image stabilization system and to one ormore components of the camera system, a synchronized movement.

Example 15: A system providing a means for performing the method asrecited by any of examples 1-9.

Example 16: A user device configured to perform the method as recited byany of examples 1-9.

Example 17: A computer-readable storage medium including instructionsthat, when executed, configure a processor to perform the method asrecited by any of examples 1-9.

What is claimed is:
 1. A method to create a super-resolution image of ascene, the method performed by a user device and comprising: detecting amotion condition, the motion condition comprising a static motioncondition or a dynamic motion condition; introducing, based on thedetected motion condition, movement to one or more components of acamera system of the user device through an optical image stabilizationsystem, the introduced movement being unsynchronized optical imagestabilization movement responsive to the motion condition being thestatic motion condition or synchronized optical image stabilizationmovement responsive to the motion condition being the dynamic motioncondition; capturing, based on the introduced movement, respective andmultiple frames of an image of the scene, the respective and multipleframes of the image of the scene having respective, sub-pixel offsets ofthe image of the scene across the multiple frames as a result of theintroduced movement to the one or more components of the camera system;performing, based on the respective, sub-pixel offsets of the image ofthe scene across the multiple frames, super-resolution computations; andcreating, based on the super-resolution computations, thesuper-resolution image of the scene.
 2. The method as recited by claim1, wherein introducing the movement to the one or more components of thecamera system is in response to entering a viewfinder mode.
 3. Themethod as recited by claim 1, wherein introducing the movement to theone or more components of the camera system is in response to receivinga capture command.
 4. The method as recited by claim 1, whereinintroducing the movement to the one or more components of the camerasystem of the user device includes introducing an in-plane movement tothe one or more components of the camera system, wherein the in-planemovement is a movement contained in a plane defined by two axes.
 5. Themethod as recited by claim 1, wherein introducing the movement to theone or more components of the camera system of the user device includesintroducing an out-of-plane movement to the one or more components ofthe camera system, wherein the out-of-plane movement is a movementoutside a plane defined by two axes.
 6. The method as recited by claim1, wherein performing the super-resolution computations includes:computing Gaussian radial basis function kernels; and computing arobustness model.
 7. The method as recited by claim 6, whereinperforming the super-resolution computations includes analyzing aninfluence of motion blur across the multiple frames.
 8. The method asrecited by claim 7, wherein performing the super-resolution computationsuses a frame-recurrent approach that uses, from the multiple frames, aprevious low-resolution frame and a current low-resolution frame tocreate a current super-resolution image.
 9. The method as recited byclaim 8, wherein additional or supplemental movement is introduced tothe one or more components of the camera system based on a coveragescore that is computed through a sampling of the multiple frames of theimage of the scene.
 10. A user device comprising: a camera system; anoptical image stabilization system; one or more processors; a display;and a computer-readable storage media storing instructions of an opticalimage stabilization system drive manager application and asuper-resolution manager application that, when executed by the one ormore processors, perform complementary functions that direct the userdevice to: detect a motion condition, the motion condition comprising astatic motion condition or a dynamic motion condition; introduce, basedon the detected motion condition, movement to one or more components ofthe camera system through the optical image stabilization system, theintroduced movement being unsynchronized optical image stabilizationmovement responsive to the motion condition being the static motioncondition or synchronized optical image stabilization movementresponsive to the motion condition being the dynamic motion condition;capture, based on the introduced movement, respective and multipleframes of an image of the scene, the respective and multiple frames ofthe image of the scene having respective, sub-pixel offsets of the imageof the scene across the multiple frames as a result of the introducedmovement to the one or more components of the camera system; perform, bythe one or more processors and based on the respective, sub-pixeloffsets of the image of the scene across the multiple frames,super-resolution computations; create, by the one or more processorsbased on the super-resolution computations, a super-resolution image ofthe scene; and render, by the display, the super-resolution image of thescene.
 11. The user device as recited by claim 10, wherein the userdevice further comprises one or more motion sensors that detect thestatic motion condition or the dynamic motion condition.
 12. The userdevice as recited by claim 10, wherein the detection of the staticmotion condition causes the user device to introduce, through theoptical image stabilization system and to one or more components of thecamera system, an unsynchronized optical image stabilization movement.13. The user device as recited by claim 10, wherein the detection of thedynamic motion condition causing the user device to introduce, throughthe optical image stabilization system and to one or more components ofthe camera system, the synchronized movement is effective to superimposethe synchronized optical image stabilization movement on a firstmovement, the superimposing being constructive or destructive.
 14. Themethod as recited by claim 10, wherein the introduction of the movementto the one or more components of the camera system is in response toreceiving a capture command.
 15. A computer-readable storage mediastoring instructions that, when executed by one or more processors,perform complementary functions that direct a user device to: detect amotion condition, the motion condition comprising a static motioncondition or a dynamic motion condition; introduce, based on thedetected motion condition, movement to one or more components of acamera system of the user device through an optical image stabilizationsystem, the introduced movement being unsynchronized optical imagestabilization movement responsive to the motion condition being thestatic motion condition or synchronized optical stabilization movementresponsive to the motion condition being the dynamic motion condition;capture, based on the introduced movement, respective and multipleframes of an image of a scene, the respective and multiple frames of theimage of the scene having respective, sub-pixel offsets across themultiple frames as a result of the introduced movement to the one ormore components of the camera system; perform, based on the respective,sub-pixel offsets of the image, super-resolution computations; andcreate, based on the super-resolution computations, a super-resolutionimage of the scene.
 16. The computer-readable storage media as recitedby claim 15, wherein the super-resolution computations include: Gaussianradial basis function kernel computations; and robustness modelcomputations that analyze an influence of motion blur across themultiple frames.
 17. The computer-readable storage media as recited byclaim 16, wherein the super-resolution computations include computationsto analyze the influence of motion blur across the multiple frames. 18.The computer-readable storage media as recited by claim 17, wherein theinstructions include instructions that direct the user device to use,from the multiple frames, a previous low-resolution frame and a currentlow-resolution frame to create a current super-resolution image.
 19. Thecomputer-readable storage media as recited by claim 15, wherein theinstructions to introduce additional or supplemental movements to theone or more components of a camera system are based on a coverage scoreof the multiple frames of the image of the scene.
 20. Thecomputer-readable storage media as recited by claim 19, wherein theinstructions include instruction to compute the coverage score based ona sampling of the multiple frames of the image of the scene.